from sklearn_benchmarks.reporting.hp_match import HpMatchReporting

scikit-learn vs. ONNX Runtime

reporting = HpMatchReporting(against_lib="onnx", config="config.yml", log_scale=True)
reporting.make_report()

We assume here there is a perfect match between the hyperparameters of both librairies. For a given set of parameters and a given dataset, we compute the speedup time scikit-learn / time onnx. For instance, a speedup of 2 means that onnx is twice as fast as scikit-learn for a given set of parameters and a given dataset.

KNeighborsClassifier_brute_force

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=brute.

WARNING! Mismatch between validation scores for 12 prediction (75.0%). The observed differences can be found in the diff_accuracy_scores column. The chosen difference threshold is 0.001 for accuracy_score. See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	parameters_digest	dataset_digest	mean_duration_sklearn	std_duration_sklearn	n_iter	...	algorithm	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	diff_accuracy_scores
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	e70d8e7886766d41f30889506baa1e67	2cf49829391aa7891e877f2ed070adf0	1.944462	0.149903	NaN	...	brute	-1	1	0.663	0.309275	0.007307	1.000	6.287163	6.288918	0.337
2	KNeighborsClassifier_brute_force	predict	100000	1	100	e70d8e7886766d41f30889506baa1e67	2cf49829391aa7891e877f2ed070adf0	0.024074	0.003250	NaN	...	brute	-1	1	1.000	17.004475	0.195576	0.757	0.001416	0.001416	0.243
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	22d5eac4a4146149b35eae6914921514	2cf49829391aa7891e877f2ed070adf0	2.760593	0.035551	NaN	...	brute	-1	5	0.757	16.894082	0.018520	0.882	0.163406	0.163406	0.125
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	16f94fc93942ff7ece860c9b0f64645f	2cf49829391aa7891e877f2ed070adf0	2.031129	0.008811	NaN	...	brute	1	100	0.882	0.315060	0.005282	1.000	6.446805	6.447711	0.118
8	KNeighborsClassifier_brute_force	predict	100000	1	100	16f94fc93942ff7ece860c9b0f64645f	2cf49829391aa7891e877f2ed070adf0	0.020564	0.000214	NaN	...	brute	1	100	1.000	16.962261	0.037067	0.757	0.001212	0.001212	0.243
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	2d5f2b2c02d77766434d9e5033b4a76c	2cf49829391aa7891e877f2ed070adf0	2.763092	0.032432	NaN	...	brute	-1	100	0.882	16.914865	0.018723	0.663	0.163353	0.163353	0.219
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	efac583623711d395ddae7a1e881ff68	2cf49829391aa7891e877f2ed070adf0	2.010532	0.001644	NaN	...	brute	1	5	0.757	0.239383	0.004843	1.000	8.398792	8.400510	0.243
14	KNeighborsClassifier_brute_force	predict	100000	1	100	efac583623711d395ddae7a1e881ff68	2cf49829391aa7891e877f2ed070adf0	0.020597	0.000252	NaN	...	brute	1	5	1.000	3.714115	0.011594	0.922	0.005546	0.005546	0.078
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	b79d63bb7670c492de5c3befac58fe29	2cf49829391aa7891e877f2ed070adf0	1.161442	0.009139	NaN	...	brute	1	1	0.663	3.759380	0.006443	0.929	0.308945	0.308946	0.266
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	e70d8e7886766d41f30889506baa1e67	88843f54689e3271092f70126e1de585	1.732155	0.015052	NaN	...	brute	-1	1	0.896	0.239641	0.004121	1.000	7.228124	7.229193	0.104
20	KNeighborsClassifier_brute_force	predict	100000	1	2	e70d8e7886766d41f30889506baa1e67	88843f54689e3271092f70126e1de585	0.004219	0.001883	NaN	...	brute	-1	1	1.000	3.727263	0.022320	0.922	0.001132	0.001132	0.078
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	22d5eac4a4146149b35eae6914921514	88843f54689e3271092f70126e1de585	2.637346	0.019484	NaN	...	brute	-1	5	0.922	3.774008	0.017433	0.896	0.698818	0.698826	0.026

12 rows × 22 columns

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.000	7.052	0.0	-1	1	17.099	0.135	0.663	0.001	0.001	See	See
3	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.000	7.057	0.0	-1	5	0.314	0.005	1.000	0.036	0.036	See	See
6	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.000	7.007	0.0	1	100	16.989	0.018	0.882	0.001	0.001	See	See
9	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.000	7.014	0.0	-1	100	0.316	0.005	1.000	0.036	0.036	See	See
12	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.013	0.004	6.127	0.0	1	5	3.768	0.012	0.896	0.003	0.003	See	See
15	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.000	7.028	0.0	1	1	0.239	0.004	1.000	0.048	0.048	See	See
18	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.004	0.000	0.377	0.0	-1	1	3.761	0.012	0.929	0.001	0.001	See	See
21	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.004	0.000	0.379	0.0	-1	5	0.237	0.003	1.000	0.018	0.018	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.944	0.150	0.000	0.002	-1	1	0.309	0.007	1.000	6.287	6.289	See	See
2	KNeighborsClassifier_brute_force	predict	100000	1	100	0.024	0.003	0.000	0.024	-1	1	17.004	0.196	0.757	0.001	0.001	See	See
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.761	0.036	0.000	0.003	-1	5	16.894	0.019	0.882	0.163	0.163	See	See
5	KNeighborsClassifier_brute_force	predict	100000	1	100	0.023	0.002	0.000	0.023	-1	5	0.315	0.006	1.000	0.074	0.074	See	See
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.031	0.009	0.000	0.002	1	100	0.315	0.005	1.000	6.447	6.448	See	See
8	KNeighborsClassifier_brute_force	predict	100000	1	100	0.021	0.000	0.000	0.021	1	100	16.962	0.037	0.757	0.001	0.001	See	See
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.763	0.032	0.000	0.003	-1	100	16.915	0.019	0.663	0.163	0.163	See	See
11	KNeighborsClassifier_brute_force	predict	100000	1	100	0.025	0.003	0.000	0.025	-1	100	0.306	0.005	1.000	0.080	0.080	See	See
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.011	0.002	0.000	0.002	1	5	0.239	0.005	1.000	8.399	8.401	See	See
14	KNeighborsClassifier_brute_force	predict	100000	1	100	0.021	0.000	0.000	0.021	1	5	3.714	0.012	0.922	0.006	0.006	See	See
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.161	0.009	0.001	0.001	1	1	3.759	0.006	0.929	0.309	0.309	See	See
17	KNeighborsClassifier_brute_force	predict	100000	1	100	0.020	0.000	0.000	0.020	1	1	0.240	0.004	1.000	0.083	0.083	See	See
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.732	0.015	0.000	0.002	-1	1	0.240	0.004	1.000	7.228	7.229	See	See
20	KNeighborsClassifier_brute_force	predict	100000	1	2	0.004	0.002	0.000	0.004	-1	1	3.727	0.022	0.922	0.001	0.001	See	See
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.637	0.019	0.000	0.003	-1	5	3.774	0.017	0.896	0.699	0.699	See	See
23	KNeighborsClassifier_brute_force	predict	100000	1	2	0.009	0.002	0.000	0.009	-1	5	0.239	0.004	1.000	0.038	0.038	See	See

KNeighborsClassifier_kd_tree

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=kd_tree.

WARNING! Mismatch between validation scores for 12 prediction (75.0%). The observed differences can be found in the diff_accuracy_scores column. The chosen difference threshold is 0.001 for accuracy_score. See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	parameters_digest	dataset_digest	mean_duration_sklearn	std_duration_sklearn	n_iter	...	algorithm	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	diff_accuracy_scores
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	fe8267e25dadb302ed498e50fabf7ef4	1eb6cf1a720a225efb91df0b529b0510	0.876317	1.276624	NaN	...	kd_tree	-1	1	0.929	2.644049	0.283099	1.000	0.331430	0.333324	0.071
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	fe8267e25dadb302ed498e50fabf7ef4	1eb6cf1a720a225efb91df0b529b0510	0.002644	0.000434	NaN	...	kd_tree	-1	1	1.000	129.771584	0.000000	0.946	0.000020	0.000020	0.054
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	dc74b969bf622bc24ba3bc62c980983b	1eb6cf1a720a225efb91df0b529b0510	1.097765	0.502636	NaN	...	kd_tree	-1	5	0.946	128.980787	0.000000	0.951	0.008511	0.008511	0.005
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	41accfcdcd5c50784bedf6164b63de99	1eb6cf1a720a225efb91df0b529b0510	5.663641	0.742520	NaN	...	kd_tree	1	100	0.951	2.630994	0.211112	1.000	2.152662	2.159581	0.049
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	41accfcdcd5c50784bedf6164b63de99	1eb6cf1a720a225efb91df0b529b0510	0.003108	0.001076	NaN	...	kd_tree	1	100	1.000	131.848519	0.000000	0.946	0.000024	0.000024	0.054
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	022f7445d43bb1dbc24dc3106c03cb93	1eb6cf1a720a225efb91df0b529b0510	3.071856	0.116202	NaN	...	kd_tree	-1	100	0.951	133.022136	0.000000	0.929	0.023093	0.023093	0.022
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	84622ba45e941db642965553529e1941	1eb6cf1a720a225efb91df0b529b0510	1.694613	0.341814	NaN	...	kd_tree	1	5	0.946	0.005019	0.000133	1.000	337.654977	337.774382	0.054
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	84622ba45e941db642965553529e1941	1eb6cf1a720a225efb91df0b529b0510	0.001532	0.000425	NaN	...	kd_tree	1	5	1.000	0.049478	0.020856	0.911	0.030964	0.033602	0.089
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	a9455565db1d8e052a783317c99744ff	1eb6cf1a720a225efb91df0b529b0510	0.947502	0.346885	NaN	...	kd_tree	1	1	0.929	0.061353	0.004726	0.894	15.443408	15.489162	0.035
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	fe8267e25dadb302ed498e50fabf7ef4	edf7cb014c2c0910e1c3aaaf8ae633f2	0.027441	0.020019	NaN	...	kd_tree	-1	1	0.891	0.005181	0.000274	1.000	5.296273	5.303700	0.109
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	fe8267e25dadb302ed498e50fabf7ef4	edf7cb014c2c0910e1c3aaaf8ae633f2	0.001961	0.000187	NaN	...	kd_tree	-1	1	1.000	0.038669	0.000424	0.911	0.050701	0.050704	0.089
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	dc74b969bf622bc24ba3bc62c980983b	edf7cb014c2c0910e1c3aaaf8ae633f2	0.022126	0.000991	NaN	...	kd_tree	-1	5	0.911	0.038170	0.000543	0.891	0.579682	0.579741	0.020

12 rows × 22 columns

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.028	0.029	0.026	0.0	-1	1	131.220	0.000	0.929	0.023	0.023	See	See
3	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.635	0.078	0.022	0.0	-1	5	2.654	0.312	1.000	1.370	1.379	See	See
6	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.588	0.084	0.022	0.0	1	100	133.774	0.000	0.951	0.027	0.027	See	See
9	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.617	0.111	0.022	0.0	-1	100	2.675	0.279	1.000	1.352	1.359	See	See
12	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.642	0.060	0.022	0.0	1	5	0.050	0.024	0.891	73.275	81.293	See	See
15	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.630	0.062	0.022	0.0	1	1	0.005	0.000	1.000	720.991	721.120	See	See
18	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.001	0.017	0.0	-1	1	0.060	0.001	0.894	0.016	0.016	See	See
21	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.000	0.032	0.0	-1	5	0.005	0.000	1.000	0.101	0.101	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.876	1.277	0.000	0.001	-1	1	2.644	0.283	1.000	0.331	0.333	See	See
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.000	0.000	0.003	-1	1	129.772	0.000	0.946	0.000	0.000	See	See
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.098	0.503	0.000	0.001	-1	5	128.981	0.000	0.951	0.009	0.009	See	See
5	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	-1	5	2.638	0.197	1.000	0.001	0.001	See	See
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	5.664	0.743	0.000	0.006	1	100	2.631	0.211	1.000	2.153	2.160	See	See
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	1	100	131.849	0.000	0.946	0.000	0.000	See	See
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	3.072	0.116	0.000	0.003	-1	100	133.022	0.000	0.929	0.023	0.023	See	See
11	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.005	0.001	0.000	0.005	-1	100	2.685	0.327	1.000	0.002	0.002	See	See
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.695	0.342	0.000	0.002	1	5	0.005	0.000	1.000	337.655	337.774	See	See
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.002	0.000	0.000	0.002	1	5	0.049	0.021	0.911	0.031	0.034	See	See
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.948	0.347	0.000	0.001	1	1	0.061	0.005	0.894	15.443	15.489	See	See
17	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.001	0.000	0.000	0.001	1	1	0.005	0.000	1.000	0.196	0.196	See	See
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.027	0.020	0.001	0.000	-1	1	0.005	0.000	1.000	5.296	5.304	See	See
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	0.000	0.002	-1	1	0.039	0.000	0.911	0.051	0.051	See	See
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.022	0.001	0.001	0.000	-1	5	0.038	0.001	0.891	0.580	0.580	See	See
23	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	0.000	0.002	-1	5	0.005	0.000	1.000	0.383	0.383	See	See

HistGradientBoostingClassifier_best

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.

WARNING! Mismatch between validation scores for 1 prediction (100.0%). The observed differences can be found in the diff_accuracy_scores column. The chosen difference threshold is 0.001 for accuracy_score. See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	parameters_digest	dataset_digest	mean_duration_sklearn	std_duration_sklearn	n_iter	...	max_leaf_nodes	min_samples_leaf	n_iter_no_change	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	diff_accuracy_scores
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	25b9f14bed7dd5d830c6ccd4dfebbf0c	8c8fffa8ec4b2d8de833421f9e32beab	0.107042	0.000942	300	...	100	100	10	0.824	0.410986	0.00975	1.0	0.260452	0.260525	0.176

1 rows × 25 columns

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	HistGradientBoostingClassifier_best	fit	100000	100000	100	93.347	0.0	300	0.001	0.001	0.501	0.021	0.824	186.379	186.541	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	0.107	0.001	300	0.007	0.0	0.411	0.01	1.0	0.26	0.261	See	See